There are a variety of ways to encode machine readable information on objects, and in particular, on printed objects. Conventional visible data carriers for printed media include various forms of bar codes, including monochrome (e.g., black and white) 1D and 2D bar codes, as well as newer higher density codes that use additional colors to carry data. One example of higher density bar codes are data glyphs, which are marks (e.g., forward and back slash marks) printed at higher resolution. When viewed from a distance, glyph codes can appear as a uniform tone, and as such, can be printed in the background around other visual information.
In these types of data carriers, the elementary units (bars or data glyph marks) are independent of other visual information and convey auxiliary data. A mark or arrangement of marks is a pattern that corresponds to an auxiliary data symbol. To read the data from a printed object, the object is first optically scanned with an image sensor, converting light to an electronic signal. The electronic signal is then analyzed to detect the elements of the mark and convert them to data.
Digital watermarking is a machine readable code in which the data is hidden within an image leveraging human visibility models to minimize visual impact on the image. For certain types of applications where image information is sparse, the auxiliary data signal can still be applied to the printed object with minimal visual impact by inserting imperceptible structures having spatial frequency and color beyond the range of human visual perception. The auxiliary data signal can be conveyed by printing ink structures or modifying existing structures with changes that are too small to see or use colors that are difficult to discern. As such, digital watermarking techniques provide the flexibility of hiding data within image content, as well as inserting data in parts of a printed object where there is little or no other visual information.
These types of visible and hidden data carriers are useful for applications where there is a need to convey variable digital data in the printed object. Hidden data carriers increase the capacity of printed media to convey visual and machine readable information in the same area. Even printed objects, or portions of objects (such as logos, pictures or graphics on a document or package) that appear identical are transformed into variable data carriers.
For some applications, it is possible to identify an image using a one-to-many pattern matching scheme. Images to be uniquely identified are enrolled in a reference database, along with metadata. In image fingerprinting schemes, image features are stored in the reference database. Then, to recognize an image, suspect images, or its features, are matched with corresponding images or features in the reference database. Once matched, the reference database can provide associated digital data stored with the image.
Data carrying signals and matching schemes may be used together to leverage the advantages of both. In particular, for applications where maintaining the aesthetic value or the information content of the image is important, a combination of digital watermarking and image fingerprinting can be used.
Combinations of watermarks and fingerprints for content identification and related applications are described in assignees U.S. Patent Publications 20060031684 and 20100322469, which are each hereby incorporated by reference in its entirety. Watermarking, fingerprinting and content recognition technologies are also described in assignee's U.S. Patent Publication 20060280246 and U.S. Pat. Nos. 6,122,403, 7,289,643, 6,614,914, and 6,590,996 which are each hereby incorporated by reference in its entirety.
In many applications, it is advantageous to insert auxiliary data in printed object in a way that does not impact the other visual information on the object, yet still enables the data to be reliably retrieved from an image captured of the object. To achieve this, a technique to exploit the gap between the limit of human visual perception and the limit of an image sensor has been developed. The gamut of human visual perception and the gamut of an image sensor are defined in terms of characteristics of the rendered output, including spatial resolution or spatial frequency and color. Each gamut is a multi-dimensional space expressed in terms of these characteristics. The gap between the gamut of human and sensor perception is a multidimensional space that our data insertion schemes exploit to insert auxiliary data without impacting other visual information on the object.
This multi-dimensional gap is a 5-dimensional space (2 spatial+3 color) or higher (spatial/color shapes, frequencies, distributions) where our methods insert:
(1) uniform texture watermarks (independent of content—but controlled for visibility), and
(2) content-based watermarks where the content is used as a reference framework. As a reference, the content is either altered in a measurable but imperceptible way or used (e.g., edges) to locate and orient an underlying variation that is intended to keep the content unchanged.
Digital printing is becoming increasingly more advanced, enabling greater flexibility and control over the image characteristics used for data insertion when preparing an image for printing. The process of preparing a digital image for printing encompasses conversion of an image by a Raster Image Processor, Raster Image Processing, halftoning, and other pre-print image processing. Background on these processes is provided below.
Along with advances in printing, the gamut of even widely used image sensors is becoming greater. For hidden data insertion, the challenge is to insert the data in the human-sensor perception gap so that it can be widely detected across many consumer devices. Of course, for certain security applications, more expensive printers and image scanners can be designed to insert security features and expand the gamut of the scanning equipment used to detect such features. This is useful to detect security features and/or tampering with such features. However, the human-device perception gap is smaller for more widely deployed sensors, such as those commonly used in mobile devices like smart phones and tablet PCs.
Our data insertion methods exploit the gap more effectively through data insertion in the process of preparing a digital image for printing. Additional control over the process of inserting auxiliary data is achieved by implementing the process in the Raster Image Processor (RIP).
A raster image processor (RIP) is a component used in a printing system which produces a raster image also known as a bitmap. The bitmap is then sent to a printing device for output. The input may be a page description in a high-level page description language such as PostScript, Portable Document Format, XPS or another bitmap of higher or lower resolution than the output device. In the latter case, the RIP applies either smoothing or interpolation algorithms to the input bitmap to generate the output bitmap.
Raster image processing is the process and the means of turning vector digital information such as a PostScript file into a high-resolution raster image. A RIP can be implemented either as a software component of an operating system or as a firmware program executed on a microprocessor inside a printer, though for high-end typesetting, standalone hardware RIPs are sometimes used. Ghostscript and GhostPCL are examples of software RIPs. Every PostScript printer contains a RIP in its firmware.
Half-toning is a process of converting an input image into halftone structures used to apply ink to a medium. The digital representation of a halftone image is sometimes referred to as a binary image or bitmap, as each elementary image unit or pixel in the image corresponds to the presence, or not, of ink. Of course, there are more variables that can be controlled at particular spatial location, such as various color components (CMYK and spot colors). Some advanced printers can control other attributes of the ink placement, such as its density or spatial depth or height.
This half-toning process is typically considered to be part of the RIP or Raster Image Processing. In some printing technologies, these halftone structures take the form of clustered dots (clustered dot half-toning). In others, the halftone structures take the form of noise-like dot patterns (e.g., stochastic screens, blue noise masks, etc.).
Our patent literature provides several techniques for digital watermarking in the halftone process. Examples of these techniques are detailed in U.S. Pat. Nos. 6,694,041 and 6,760,464, which are each hereby incorporated herein by reference in its entirety.
New printing techniques enable very fine structures to be created in the RIP which will appear visually identical to the eye. For example a 50% gray can be created with a conventional clustered dot screen pattern at 150 lines per inch, or exactly the same visual effect can be created with a much higher frequency line structure such as a stochastic screen. Usually, these two structures are not mixed on one page, as they have very different dot gain characteristics and require different corrections. However, our methods are able to correct for the mechanical dot gain, so that the two patterns appear identical when they appear on the same page. See, in particular, our prior work in dot gain correction, printer calibration, and compensating for printer and scanner effects, in U.S. Pat. Nos. 6,700,995, 7,443,537, and U.S. Patent Publication 20010040979, which are each hereby incorporated herein by reference in its entirety.
Mobile devices have a capture resolution of much greater than 150 lpi (resolution of newer phones, such as iPhone 4 is about 600 lpi or better), so they can be used to distinguish between these two types of patterns. One particular example is an image that appears as a uniform texture, yet a watermark pattern is inserted into it by modulating the line screen frequency and direction according to a watermark signal pattern. In particular, the locations of a watermark pattern are printed using a higher frequency line pattern at first direction (e.g., vertical screen angle). The other locations are printed with a lower frequency line pattern in another direction (e.g., diagonal screen angle). The watermark signal is modulated into the image by selection of a higher frequency screen at an arrangement of spatial locations that form the watermark signal pattern. When printed, these locations look similar to surrounding locations. However, when scanned, the sensor sees these locations as being different, and the watermark pattern in the resulting electronic image is easier to detect.
This approach allows a whole set of new messaging techniques to be used in the range between 150 lpi and 600 lpi where 2 spatial dimensions and 3 dimensions of color information can be inserted. This information can be watermark, barcode or any other signaling mechanism.